55 research outputs found
How Do Deepfakes Move? Motion Magnification for Deepfake Source Detection
With the proliferation of deep generative models, deepfakes are improving in
quality and quantity everyday. However, there are subtle authenticity signals
in pristine videos, not replicated by SOTA GANs. We contrast the movement in
deepfakes and authentic videos by motion magnification towards building a
generalized deepfake source detector. The sub-muscular motion in faces has
different interpretations per different generative models which is reflected in
their generative residue. Our approach exploits the difference between real
motion and the amplified GAN fingerprints, by combining deep and traditional
motion magnification, to detect whether a video is fake and its source
generator if so. Evaluating our approach on two multi-source datasets, we
obtain 97.17% and 94.03% for video source detection. We compare against the
prior deepfake source detector and other complex architectures. We also analyze
the importance of magnification amount, phase extraction window, backbone
network architecture, sample counts, and sample lengths. Finally, we report our
results for different skin tones to assess the bias
My Face My Choice: Privacy Enhancing Deepfakes for Social Media Anonymization
Recently, productization of face recognition and identification algorithms
have become the most controversial topic about ethical AI. As new policies
around digital identities are formed, we introduce three face access models in
a hypothetical social network, where the user has the power to only appear in
photos they approve. Our approach eclipses current tagging systems and replaces
unapproved faces with quantitatively dissimilar deepfakes. In addition, we
propose new metrics specific for this task, where the deepfake is generated at
random with a guaranteed dissimilarity. We explain access models based on
strictness of the data flow, and discuss impact of each model on privacy,
usability, and performance. We evaluate our system on Facial Descriptor Dataset
as the real dataset, and two synthetic datasets with random and equal class
distributions. Running seven SOTA face recognizers on our results, MFMC reduces
the average accuracy by 61%. Lastly, we extensively analyze similarity metrics,
deepfake generators, and datasets in structural, visual, and generative spaces;
supporting the design choices and verifying the quality.Comment: 2023 IEEE Winter Conference on Applications of Computer Vision (WACV
How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting Residuals with Biological Signals
Fake portrait video generation techniques have been posing a new threat to
the society with photorealistic deep fakes for political propaganda, celebrity
imitation, forged evidences, and other identity related manipulations.
Following these generation techniques, some detection approaches have also been
proved useful due to their high classification accuracy. Nevertheless, almost
no effort was spent to track down the source of deep fakes. We propose an
approach not only to separate deep fakes from real videos, but also to discover
the specific generative model behind a deep fake. Some pure deep learning based
approaches try to classify deep fakes using CNNs where they actually learn the
residuals of the generator. We believe that these residuals contain more
information and we can reveal these manipulation artifacts by disentangling
them with biological signals. Our key observation yields that the
spatiotemporal patterns in biological signals can be conceived as a
representative projection of residuals. To justify this observation, we extract
PPG cells from real and fake videos and feed these to a state-of-the-art
classification network for detecting the generative model per video. Our
results indicate that our approach can detect fake videos with 97.29% accuracy,
and the source model with 93.39% accuracy.Comment: To be published in the proceedings of 2020 IEEE/IAPR International
Joint Conference on Biometrics (IJCB
My Art My Choice: Adversarial Protection Against Unruly AI
Generative AI is on the rise, enabling everyone to produce realistic content
via publicly available interfaces. Especially for guided image generation,
diffusion models are changing the creator economy by producing high quality low
cost content. In parallel, artists are rising against unruly AI, since their
artwork are leveraged, distributed, and dissimulated by large generative
models. Our approach, My Art My Choice (MAMC), aims to empower content owners
by protecting their copyrighted materials from being utilized by diffusion
models in an adversarial fashion. MAMC learns to generate adversarially
perturbed "protected" versions of images which can in turn "break" diffusion
models. The perturbation amount is decided by the artist to balance distortion
vs. protection of the content. MAMC is designed with a simple UNet-based
generator, attacking black box diffusion models, combining several losses to
create adversarial twins of the original artwork. We experiment on three
datasets for various image-to-image tasks, with different user control values.
Both protected image and diffusion output results are evaluated in visual,
noise, structure, pixel, and generative spaces to validate our claims. We
believe that MAMC is a crucial step for preserving ownership information for AI
generated content in a flawless, based-on-need, and human-centric way
Topology-Aware Loss for Aorta and Great Vessel Segmentation in Computed Tomography Images
Segmentation networks are not explicitly imposed to learn global invariants
of an image, such as the shape of an object and the geometry between multiple
objects, when they are trained with a standard loss function. On the other
hand, incorporating such invariants into network training may help improve
performance for various segmentation tasks when they are the intrinsic
characteristics of the objects to be segmented. One example is segmentation of
aorta and great vessels in computed tomography (CT) images where vessels are
found in a particular geometry in the body due to the human anatomy and they
mostly seem as round objects on a 2D CT image. This paper addresses this issue
by introducing a new topology-aware loss function that penalizes topology
dissimilarities between the ground truth and prediction through persistent
homology. Different from the previously suggested segmentation network designs,
which apply the threshold filtration on a likelihood function of the prediction
map and the Betti numbers of the ground truth, this paper proposes to apply the
Vietoris-Rips filtration to obtain persistence diagrams of both ground truth
and prediction maps and calculate the dissimilarity with the Wasserstein
distance between the corresponding persistence diagrams. The use of this
filtration has advantage of modeling shape and geometry at the same time, which
may not happen when the threshold filtration is applied. Our experiments on
4327 CT images of 24 subjects reveal that the proposed topology-aware loss
function leads to better results than its counterparts, indicating the
effectiveness of this use
Generative Street Addresses from Satellite Imagery
We describe our automatic generative algorithm to create street addresses from satellite images by learning and labeling roads, regions, and address cells. Currently, 75% of the world’s roads lack adequate street addressing systems. Recent geocoding initiatives tend to convert pure latitude and longitude information into a memorable form for unknown areas. However, settlements are identified by streets, and such addressing schemes are not coherent with the road topology. Instead, we propose a generative address design that maps the globe in accordance with streets. Our algorithm starts with extracting roads from satellite imagery by utilizing deep learning. Then, it uniquely labels the regions, roads, and structures using some graph- and proximity-based algorithms. We also extend our addressing scheme to (i) cover inaccessible areas following similar design principles; (ii) be inclusive and flexible for changes on the ground; and (iii) lead as a pioneer for a unified street-based global geodatabase. We present our results on an example of a developed city and multiple undeveloped cities. We also compare productivity on the basis of current ad hoc and new complete addresses. We conclude by contrasting our generative addresses to current industrial and open solutions. Keywords: road extraction; remote sensing; satellite imagery; machine learning; supervised learning; generative schemes; automatic geocodin
- …